TITLE OF THE INVENTION 



AUTOMATED METHOD AND SYSTEM FOR COMPUTERIZED IMAGE ANALYSIS 

FOR PROGNOSIS 

CROSS-REFERENCE TO RELATED APPLICATIONS 
The present application claims priority to the earlier filing date of provisional U.S. 
Application No. 60/395,305, filed July 12, 2002, the entire contents of which are incorporated 
herein by reference. 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

The present invention relates generally to systems and methods for computer-aided 
diagnosis in the assessment of a prognosis for cancer patients. 

The present invention also generally relates to automated techniques for the detection 
of abnormal anatomic regions, for example, as disclosed, in particular, in one or more of U.S. 
Patent Nos. 4,907,156; 5,133,020; 5,832,103; and 6,138,045; all of which are incorporated 
herein by reference. 

The present invention also generally relates to computerized techniques for automated 
analysis of digital images, for example, as disclosed in one or more of U.S. Patent Nos. 
4,839,807; 4,841,555; 4,851,984; 4,875,165; 4,918,534; 5,072,384; 5,150,292; 5,224,177; 
5,289,374; 5,319,549; 5,343,390; 5,359,513; 5,452,367; 5,463,548; 5,491,627; 5,537,485; 
5,598,481; 5,622,171; 5,638,458; 5,657,362; 5,666,434; 5,673,332; 5,668,888; 5,732,697; 
5,740,268; 5,790,690; 5,873,824; 5,881,124; 5,931,780; 5,974,165; 5,982,915; 5,984,870; 
5,987,345; 6,011,862; 6,058,322; 6,067,373; 6,075,878; 6,078,680; 6,088,473; 6,112,112; 
6,141,437; 6,185,320; 6,205,348; 6,240,201; 6,282,305; 6,282,307; 6,317,617 as well as U.S. 
patent applications 08/173,935; 08/398,307 (PCT Publication WO 96/27846); 08/536,149; 
08/900,189; 09/027,468; 09/141,535; 09/471,088; 09/692,218; 09/716,335; 09/759,333; 
09/760,854; 09/773,636; 09/816,217; 09/830,562; 09/818,831; 09/842,860; 09/860,574; 
60/160,790; 60/176,304; 60/329,322; 09/990,311; 09/990,310; 09/990,377; 10/360,814; and 
60/331,995; and PCT patent applications PCT/US98/15165; PCT/US98/24933; 
PCT/US99/03287; PCT/US00/41299; PCT/US0 1/00680; PCT/US01/01478 and 
PCT/US0 1/0 1479, all of which are incorporated herein by reference. 
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The present invention includes the use of various technologies referenced and 
described in the above-noted U.S. Patents, as well as described in the references identified in 
the following LIST OF REFERENCES by the author(s) and year of publication and cross- 
referenced throughout the specification by reference to the respective number in parentheses, 
of the reference: 
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The entire contents of each related patent listed above and each reference listed in the 
LIST OF REFERENCES, are incorporated herein by reference. 

DISCUSSION OF THE BACKGROUND 

Breast cancer remains a disease without a cure unless the cancer is found at a 
sufficiently early stage and subsequently surgically removed, irradiated, or eradicated with 
chemotherapy. Major research issues include those focused on genetic and molecular forms 
of detection and treatment; and those focused on anatomical levels of prevention, detection, 
and treatment. In these various areas, the role of the human interpreter (e.g., oncologist, 
radiologist, pathologist, surgeon, primary care physician) varies. However, the very presence 
of a human interpreter introduces subjective judgment into the decision-making process — 
whether it be in the initial detection (or mis-detection) of a lesion on a mammogram, or in the 
surgical decision regarding the type of incision. Thus, while ongoing research is needed to 
understand the biological aspects of cancer, to improve the physical aspects of 
instrumentation to better "see" the cancer, and to improve the biological/chemical/physical 
aspects of therapy, research is also needed to improve the human role in the overall 
management of the patient. Multi-modality and multi-disciplinary decision making on patient 
management, requiring inputs from oncologists, pathologists, radiologists, surgeons, and risk 
clinic physicians, can be quite subjective, as is often evident during case management 



6 



conferences. Although "subjective" does not necessarily mean "poor judgement," it does 
allow sub-optimal and inconsistent decision making. 

Breast cancer is the leading cause of death for women in developed countries. Early 
detection of breast cancer increases the success of treatment dramatically. Thus, screening for 
breast cancer of women over 40 years of age is generally recommended. Current methods for 
detecting and diagnosing breast cancer include mammography, sonography (also referred to 
as ultrasound), and magnetic resonance imaging (MRI). 

Mammography is the most effective method for the early detection of breast cancer. 
Periodic screening of asymptomatic women has been shown to reduce mortality [1-6]. Many 
breast cancers are detected and referred for surgical biopsy on the basis of a radiographically 
detected mass lesion or cluster of microcalcifications. Although general rules for the 
differentiation between benign and malignant mammographically identified breast lesions exist, 
considerable misclassification of lesions occurs with the current methods. On average, less than 
30% of masses referred for surgical breast biopsy are actually malignant. 

Currently, computerized analysis schemes are being developed to aid in distinguishing 
between malignant and benign lesions in order to improve both sensitivity and specificity. 
Comprehensive summaries of investigations in the field of mammography CAD have been 
published by Giger and colleagues [7-9]. Investigators have used computers to aid in the 
decision-making process regarding likelihood of malignancy and patient management using 
human-extracted features and BI-RADS [10-13]. Such methods are dependent on the subjective 
identification and interpretation of the mammographic data by human observers. 

Gale et al. [14] and Getty et al. [15] each developed computer-based classifiers, which 
take as input diagnostically relevant features obtained by the reading of breast images by 
radiologists. Getty et al. found that with the aid of the classifier, community radiologists 
performed as well as unaided expert mammographers in making benign-malignant decisions. 
Swett et al. [16] developed an expert system to provide visual and cognitive feedback to the 
radiologist using a critiquing approach combined with an expert system. Other investigators 
have developed methods based on computer-extracted features [17-24]. The benefit of using 
computer-extracted features is the objectivity and reproducibility of the result. Radiologists 
employ many radiographic image features, which they seem to extract and interpret 
simultaneously and instantaneously. Thus, the development of methods using computer- 
extracted features requires, besides the determination of which individual features are clinically 
significant, the computerized means for the extraction of each such feature. 
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Spatial features, which are characteristic of lesions, have been shown to be extractable 
by a computer analysis of the mammograms, and to be useful in distinguishing between 
malignant and benign lesions. Most methods are evaluated in terms of their ability to 
distinguish between malignant and benign lesions, although a few have been evaluated in 
terms of patient management (i.e., return to screening vs. biopsy). It is important to state that 
while one of the aims of computerized classification is to increase sensitivity, another is to 
reduce the number of benign cases sent for biopsy. Such a reduction will be clinically 
acceptable only if it does not result in un-biopsied malignant cases, however, since the "cost" 
of a missed cancer is much greater than misclassification of a benign case. Thus, computer 
classification schemes should be developed to improve specificity, but not at the loss of 
sensitivity. It has been shown that the computerized analysis of mass lesions [17] [21] and 
clustered microcalcifications [18] [22] on digitized mammograms yields performances similar 
to an expert mammographer, and significantly better than average radiologists in the task of 
distinguishing between malignant and benign lesions. 

Giger et al. [35-40] have been successful in developing mathematical descriptors of 
breast lesions in mammography, ultrasound, and MRI for the task of estimating the likelihood 
of malignancy. The computerized classification scheme for mass lesions is based on the 
degree of spiculation exhibited by the mass in question. Various features related to the 
margin, shape, and density of each mass can also be extracted automatically from the 
neighborhoods of the computer-extracted mass regions. Similarly, computer-extracted 
features in sonography of the breast as well as both spatial and temporal lesion characteristics 
in dynamic MRI of the breast have been investigated. Selected features were merged into an 
estimated likelihood of malignancy by use of an artificial neural network. Evaluation was 
performed using independent test sets and ROC analysis. Results from an independent 
evaluation of 1 10 cases indicated that the classification scheme is robust to the variations in 
case-mix and digitization techniques. In an observer study including six mammographers and 
six community general radiologists using the independent database, use of computer results as 
an aid yielded significant improvement in their performance. Similar results were found for 
ultrasound CAD in an observer study. While these results we obtained in the task of 
characterizing malignant and benign lesions, similar features may be used to characterize 
cancer of various appearances. 

Further, Giger et al. [41] [42] have developed computerized methods for characterizing 
"normal" breast structure that may be used together with clinical measures of risk for use in 
quantitatively characterizing the breast parenchyma and risk of cancer. In initial studies, three 
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different approaches were investigated to relate mammographic patterns, as characterized by 
computer-extracted features, to the risk of developing breast cancer, using three high-risk groups: 
(a) BRCA1/BRCA2 mutation carriers, (b) women with unilateral breast cancer, and (c) women 
with a strong family history of breast/ovarian cancer and/or with a history of previous breast 
biopsy for benign diseases; and a low-risk group including women with no family history of 
breast/ovarian cancer and no prior history of breast cancer and benign breast disease. 
Quantitative analyses of mammographic patterns, from all three groups, indicated that women at 
high risk tend to have dense breast with coarse and low-contrast texture patterns. While these 
computer-extracted features were developed to help determine risk of breast cancer, these 
features and derivatives and extensions of them may be used to characterize the parenchyma in 
general in terms of density and texture. 

With respect to prognosis, histological features of the primary tumor have been shown 
to predict prognosis in terms of early distant relapse in node-negative breast cancer patients 
[26]. Tumor progression was shown to be correlated with increasing tumor size and nodal 
involvement [27][33]. In addition, various biomarkers have been investigators as prognostic 
factors [28] [29]. Also, visual mammographic classification seems to predict good and bad 
long-term outcomes for survival in tumors of 14 mm or smaller [30][31]. Others have also 
investigated the use of clinical and mammographic appearance of tumors as a means to 
predict the risk of local recurrence [32]. 

SUMMARY OF THE INVENTION 
Accordingly, an object of this invention is to provide a method and system that 
analyzes abnormalities and parenchyma regions in medical images to yield information on 
prognosis. 

Another object of this invention is to provide a method and system that distinguishes 
between different types of cancerous lesions in terms of their severity and prognosis. This 
can be achieved by varying the "truth" in the training of the computer system. 

These and other objects are achieved according to the invention by providing a new 
automated method and system that analyzes lesions on medical images in which the output of 
the analysis relates to the prognosis. 

Accordingly, the present invention is directed to an automated, computerized method 
for the prognosis of lesions. The techniques include (1) segmentation of lesions from 
radiographic images, (2) extraction of lesion features, and (3) a merging of the features (with 
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and without clinical information), to yield as estimate of the prognosis for the specific case. 
An example is given for the prognosis of breast cancer lesions on mammograms. 

According to one aspect of the present invention, there is provided a method, system, 
and computer program product for processing medical image data to determine a prognosis 
on recovery, comprising: (1) obtaining segmented image data of a portion of the medical 
image data corresponding to an abnormality; (2) extracting at least one abnormality feature 
from the segmented image data corresponding to the abnormality; and (3) determining the 
prognosis on recovery based on the extracted at least one abnormality feature. 

According to one aspect of the present invention, the step of extracting at least one 
abnormality feature comprises determining a radial gradient index. 

According to another aspect of the present invention, the step of extracting at least one 
abnormality feature comprises determining at least one of circularity and density of the 
abnormality. 

According to still another aspect of the present invention, the step of extracting at 
least one abnormality feature comprises determining at least one of average gray level, 
contrast, and a texture measure of the abnormality. 

According to still another aspect of the present invention, the determining step 
comprises applying the extracted at least one abnormality feature to an artificial neural 
network (ANN) which classifies the abnormality at an output of the ANN. 

According to still another aspect of the present invention, the determining step 
comprises applying the extracted at least one abnormality feature to a linear discriminant 
which classifies the abnormality at an output of the linear discriminant. 

According to another aspect of the present invention, there is provided a method, 
system, and computer program product for processing medical image data to determine a 
prognosis on recovery, comprising: (1) obtaining segmented image data of a portion of the 
medical image data corresponding to a parenchyma region; (2) extracting at least one 
parenchyma feature from the segmented image data corresponding to the parenchyma region; 
and (3) determining the prognosis based on the extracted at least one parenchyma feature. 

According to still another aspect of the present invention, there is provided a method, 
system, and computer program product for processing medical image data to determine a 
prognosis on recovery, comprising: (1) obtaining segmented image data of a portion of the 
medical image data corresponding to an abnormality; (2) extracting at least one abnormality 
feature from the segmented image data corresponding to the abnormality; (3) obtaining 
segmented image data of a portion of the medical image data corresponding to a parenchyma 



10 



region; (4) extracting at least one parenchyma feature from the segmented image data 
corresponding to the parenchyma region; and (5) determining the prognosis on recovery based 
on the extracted at least one abnormality feature and the extracted at least one parenchyma 
feature. 

Further, the present invention is embodied in a computerized image analysis system 
for assessing prognosis that combines the computerized analysis of medical images of 
abnormalities (e.g., cancerous lesions) with the training-based method of assessing prognosis 
of a patient, such as lymph node involvement, presence of metastatic disease, local 
recurrence, and/or death. It is expected that use of such a system to assess the severity of the 
disease will aid in improved decision making regarding treatment options. 

BRIEF DESCRIPTION OF THE DRAWINGS 
A more complete appreciation of the invention and many of the attendant advantages 
thereof will be readily obtained as the same becomes better understood by reference to the 
following detailed description when considered in connection with the accompanying 
drawings, wherein: 

Figure 1 illustrates a specific example (breast cancer prognosis) of the method of 
computerized image analysis for assessment of a prognosis according to the present 
invention; 

Figure 2 shows (a) an illustration defining the radial angle as the angle between the 
direction of the maximum gradient and its radial direction, and a demonstration of the 
normalized cumulated edge-gradient distributions for (b) speculated, and (c) circular masses 
(FWHM is extracted from such plots to yield the spiculation measure); 

Figure 3 is a diagram of the general system for computerized image analysis for 
assessment of a prognosis according to the present invention; 

Figure 4 illustrates the general method of computerized image analysis for assessment 
of a prognosis according to the present invention; 

Figure 5 illustrates the method of training a classifier for performing computerized 
image analysis for assessment of a prognosis according to the present invention; 

Figure 6 shows a computer-extracted mammographic lesion with the computer- 
determined margin in outline; 

Figure 7 shows an example of the computer analysis for a node-positive case and a 
node-negative case; 
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Figure 8 shows the relationship between shape from radial gradient analysis and 
texture within the lesions for lesions from node-positive and node-negative breast cancer 
cases; 

Figure 9 shows the relationship between shape from radial gradient analysis and 
normalized texture within the lesions for lesions from node-positive and node-negative breast 
cancer cases; 

Figure 10 shows a histogram illustrating the distribution of node-positive and node- 
negative cases in terms of the output from linear discriminant analysis; 

Figure 1 1 shows ROC curves illustrating the performance of the system in 
distinguishing between cases with good and poor prognosis (in terms of nodal involvement); 
and 

Figure 12 is a diagram of the system for training a classifier to perform computerized 
image analysis for assessment of a prognosis according to the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Due to the subjective nature of human interpretation of mammographic images for 
diagnosis or prognosis, a method for assessing prognosis based on a computer analysis of the 
image of the lesion has been developed. Similar lesion features as those for diagnosis may be 
extracted, although the analysis of the features and the training of the computer system 
requires different truth. For prognosis, that truth can be lymph node involvement, presence of 
metastatic disease, local recurrence, and/or death. 

Lymph node status is a useful marker for assessing the extent of diagnosed breast 
cancer. The more extended, the poorer the prognosis. Lymph node status influences the 
choice of treatment. Thus, in the particular example presented here, lymph node involvement 
is used as the truth. 

The present invention involves the use of computer-aided diagnosis as an aid to 
radiologists in the assessment of the cancerous lesion on mammography. The mass analysis 
method typically includes three components: (1) automated segmentation of mass regions; (2) 
automated feature-extraction; and (3) automated assessment of prognosis. The method is 
trained with truth of patients, such as lymph node involvement, presence of metastatic 
disease, local recurrence, and/or death. 

A method and system that employs a lesion assessment module for assessing the 
prognosis is presented below. The specific example presented here includes a computerized 
method for the characterization of mammographic lesions for prognosis based on image 
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analysis of mammograms. Using a combination of the lesion features can result in improved 
assessment of prognosis. A computerized image analysis system for assessing prognosis 
combines the computerized analysis of medical images of cancerous lesions with training 
based on methods of assessing prognosis of a patient, such as lymph node involvement, 
presence of metastatic disease, local recurrence, and/or death. It is expected that the use of 
such a system to assess the severity of the disease will aid in improved decision-making 
regarding treatment options. 

Referring now to the drawings, and more particularly to Figure 1 thereof, there is 
shown a schematic diagram of the method that takes as input a mammogram and outputs the 
assessment of prognosis for a specific disease. In the various methods for implementing this 
technique, various features either manually determined or extracted by computer are used. In 
addition, various truths may be used individually or combined to train a computerized image 
analyzer for the assessment of prognosis. These truths include lymph node involvement, 
presence of metastatic disease, presence of local recurrence, and death. 

In step 101, digital mammographic data is obtained. As will be described in more 
detail below, at least one lesion is located and segmented in step 102. Next, in step 103, 
various mathematical features of the lesion are extracted from the segmented lesion identified 
in step 102. Additionally, in step 104, parenchyma features are also extracted from the 
mammogram data. As described below, the parenchyma features are obtained from a 
parenchyma region distinct from that of the segmented lesion in the mammogram. 
Alternatively, a second mammogram may be used to obtain the parenchyma features. 

In step 105, the lesion and parenchyma features are combined using classifier, such as 
a neural network or linear discriminant, to produce a numerical indication of the prognosis of 
recovery. Alternatively, the inputs to the classifier may be only the lesion features or only the 
parenchyma features. In step 106, an assessment of the prognosis, including, e.g., the 
likelihood of survival or suggested treatment is displayed. Step 107 is directed to the a priori 
training of the classifier using truth indicators such as lymph node status. 

Figure 3 illustrates the system for determining prognosis on recovery according to the 
present invention. The medical image is acquired by the medical image acquisition unit 301 
and converted, if necessary, to digital format by the AID and Discrete Sampling Unit 302. 
The digitized image is then processed by the lesion location unit 303 and the lesion extractor 
unit 304. Region growing is performed by the lesion extractor 304 to extract at least one 
lesion, including its margin. After lesion extraction, the lesion feature extractor 305 
computes numerical descriptions of lesion features, described below. The numerical 
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descriptors of the lesion are then classified by the already-trained classifier 306, which may 
be an ANN or a linear discriminant. 

Additionally, the parenchyma location unit 308 locates a parenchyma region within 
the medical image acquired by the medical image acquisition unit 301. Note that the 
parenchyma region may alternatively be located in a different medical image than the one 
processed by the lesion locator unit. For example, an image of the contralateral breast may be 
used to locate the parenchyma region in a breast cancer prognosis determination. 

The parenchyma feature extraction unit 309 calculates parenchyma feature values 
from the pixel values in the parenchyma region obtained by the parenchyma location unit 308. 
The parenchyma feature values also input into the trained classifier 306. Note that, in 
addition to the computer-extracted features, clinical data, such as patient age, can also be 
incorporated in the classifier as additional features. 

In an alternative embodiment, only the lesion features output from the lesion feature 
extractor 305 are input into the trained classified 306. In a second alternative embodiment, 
only the parenchyma features output from the parenchyma feature extraction unit 309 are 
input into the trained classified 306. 

The output of the classifier 306 is given as an assessment of a prognosis to the display 
307. The numerical value output by the classifier 306 can be related to the likelihood of 
survival or the likelihood of good prognosis, etc. 

Figure 4 illustrates the method of determining prognosis on recovery according to the 
present invention. In step 401, at least one medical image (e.g., sonogram, radiograph, CT 
scan, or MRI) is acquired. The at least one medical image obtained in step 401 is digitized in 
step 402. Alternatively, step 402 may be unnecessary if digitized image data is obtained in 
step 401, for example, from a memory device or digitized image database. For example, the 
screen/film mammogram would need to be digitized, but a thoracic CT image is already in a 
digital format. 

In step 403, an abnormality is located within the at least one digitized medical image 
obtained in step 402. Step 403 may be done automatically by computer or manually. 

Next, in step 404, the abnormality located in step 403 is extracted and segmented 
using region-growing techniques, as described in more detail below. 

In step 405, various mathematical feature values of the abnormality are extracted, 
using the methods described below. Feature extraction involves various computations of 
mathematical descriptors of abnormality features, examples of which include texture and 
RGI. 
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In step 406, a parenchyma region within the at least one medical image is obtained. 
Note that the parenchyma region may be located within a different medical image from the 
image from which the abnormality is extracted. 

Next, in step 407, various parenchyma features are extracted from the parenchyma 
region obtained in step 406. Parenchyma features are described in more detail below. 

In step 408, the parenchyma and abnormality features calculated in steps 405 and 407 
are combined using a classifier (ANN or LDA) to obtain a numerical indication of prognosis. 
Alternatively, the inputs to the classifier may be only the lesion features or only the 
parenchyma features. The numerical indication is related to the likelihood of survival or the 
likelihood of good prognosis, etc. In step 409, the numerical indication of prognosis is 
converted into a formal prognosis including likelihood of survival and treatment 
recommendations. The prognosis is then displayed in step 410. 

Figure 5 illustrates the preferred method of training the prognosis determination 
system of the present invention. Steps 401-409 shown in Figure 5 are identical to the 
similarly numbered steps shown in Figure 4, except that in the training process, known truth 
indicators are associated with the medical image obtained in step 401. In step 510, the 
numerical indication of prognosis is compared to one or more truth indicators (death, lymph 
node status, etc.) to evaluate the performance of the classifier (step 408). ROC analysis may 
be used in step 510. Based on this comparison, the classifier is modified. For example, the 
weights of a neural network may be adjusted based on the comparison of the truth indicators 
and the classifier output. The steps shown in Figure 5 may be repeated for additional training 
images. The training process is described in more detail below. 

Figure 12 illustrates the system for training the prognosis determination system of the 
present invention. Note that Figure 12 includes the overall prognosis determination system 
shown in Figure 3, but also includes the Classifier Training Unit 1201, which modifies the 
classifier 306 based on the output of the classifier 306 and the truth indicators described 
above. According the system of Figure 12 is used to implement the method described with 
regard to Figure 5. 

Lesion and Parenchyma Features 

Radiographically, mass lesions can be characterized by their degree of spiculation, 
margin definition, shape, density, homogeneity (texture), asymmetry, and temporal stability 
[7] [9]. The method for the characterization of mass lesions from mammograms is based on 
earlier work [17][21][23][24] in which a characterization scheme based on the degree of 
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spiculation, as determined from a cumulative edge gradient histogram analysis in which the 
gradient is analyzed relative to the radial angle (Figure 2), was developed. Features related to 
the margin, shape, and density of each mass are extracted automatically from the image data 
and merged into an estimate of the likelihood of malignancy [17] [21] [2] [24]. These features 
include a spiculation measure (Figure 2), a margin definition feature, and three density 
measures. 

The mass is first extracted from the anatomic background of the mammogram using 
automatic region-growing techniques [17]. Features extracted are then obtained using 
cumulative edge gradient histogram analysis. In the cumulative edge-gradient analysis, the 
maximum gradient and angle of this gradient relative to the radial direction is calculated. 
Note that here the spiculation feature (based on the radial direction) is used in distinguishing 
between spiculated lesions and round lesions. Also, the average gradient along the margin of 
a mass will be calculated to describe the sharpness of the margin. Higher values indicate a 
sharper margin, and thus a higher likelihood that the lesion is benign. 

In addition, a radial gradient index (normalized radial gradient) that describes the 
circularity and density characteristics of a lesion is used and is given by 



Here RGI is the radial gradient index (-1< RGI <1); P is an image point; L is a lesion 
excluding the center part; D x is the gradient in x-direction; Dy is the gradient in y-direction; 
and cp is the angle between the gradient vector and the connection line from the center point 
to a neighbor point. 

Although the radiographic density of a mass may not be by itself as powerful a 
prognostic index, taken with other features, density assessment may be extremely useful. The 
evaluation of the density of a mass may be of particular importance in assessing lesions that 
are not spiculated. 

In order to assess the density of a mass radiographically, three density-related 
measures (average gray level, contrast, and texture measure) that characterize different 
aspects of the density of a mass are used. These measures are similar to those used intuitively 
by radiologists. Average gray level is obtained by averaging the gray level values of each 
point within the grown region of a mass. Contrast is the difference between the average gray 




P<=L 



16 



level of the grown mass and the average gray level of the surrounding fatty areas (areas with 
gray-level values in the lower 20% of the histogram for the total surrounding area). Texture 
is defined here as the standard deviation of the average gradient within a mass and it is used 
to quantify patterns arising from veins, trabeculae, and other structures that may be visible 
through a low-density mass, but not through a high-density mass. A mass of low radiographic 
density should have low values of average gray level and contrast, and a high value of the 
texture measure, whereas a mass of high radiographic density should have high values of 
average gray level and contrast, and a low value of the texture measure. 

In the present invention, computer-extracted mammographic features such as 
skewness and RMS variation from Fourier analysis are used to characterize denseness, and 
the mathematical descriptors of coarseness and contrast are used to characterize the 
heterogeneity of the parenchymal texture patterns in the breast [41] [42]. A dense region of 
interest (ROI) tends to have more pixels with high gray-level values (low optical density), 
yielding a gray-level histogram skewed to the left. A fatty ROI tends to have more pixels 
with low gray-level values (high optical density), yielding a gray-level histogram skewed to 
the right. Features such as skewness and balance (defined below) of a histogram relative to 
the mean can be used to quantify the ratio of pixels with high gray-level values to those with 
low gray-level values relative to the mean, thereby approximating the local tissue 
composition (e.g., fibroglandular tissue vs. fat). A dense ROI should yield a negative value of 
skewness and a value greater than one for balance, whereas a fatty ROI should yield a positive 
value of skewness and a value less than one for balance. The skewness measure has been 
studied by Byng et al. [43] to evaluate percent mammographic density in the breast. A related 
balance measure has been studied by Tahoces et al. [44] to classify mammographic patterns 
into the Wolfe pattern. 

In the present invention, skewness is calculated using the second and third moments 
of the gray-level distribution of pixels in an ROI: 



SKEWNESS = 
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and rii is the number of occurrences of gray- level value i . G h is the highest gray-level value 
in the ROL 

Coarseness and contrast were used to characterize Wolfe pattern by Tahoces et al. 
[44]. The mathematical definitions of these two texture features are given below. The 
coarseness of a texture is defined by the amount of local variation in gray level. The contrast 
of a texture is defined by the amount of differences among all gray levels in the ROI and the 
amount of local variation in gray level presented in the ROI. Note that the contrast measure is 
determined by two terms: the gray-level differences in an ROI weighted by the amount of 
local variation. Thus, ROIs that have similar gray level differences may have different 
contrast depending on the local variation in the ROIs. Conversely, ROIs that have the same 
amount of local variation may have different contrast depending on the gray level differences 
in the ROIs. In the present invention, coarseness (local uniformity) is given by: 

COARSE = [tp t s(j)Y x 

i 

Contrast (local contrast) is given by: 

CON = [ 1 | G ±p iPj {i - jf \- 2 h(0] 

N g (N g — 1) ; = 0y=0 J H i=0 

where N g is the total number of different gray levels present in the ROI, G h is the highest 
gray-level value in the ROI, p t is the probability of occurrence of gray-level value / , N is the 
width of the ROI, d is the neighborhood size (half of the operating kernel size), n = TV - 2d , 
and the /th entry of s is given by: 
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While the examples above are described for two dimensional images, it will be 
recognized by those skilled in the art that analogous features may be calculated directly in 
three dimensions for images containing data in three or more dimensions (e.g. ultrasound, 
MRI, or CH. 

Classifiers 

According to a preferred embodiment of the present invention, linear discriminant 
analysis (LDA) and artificial neural networks (ANNs) are used to merge features into a 
prognostic marker. A discriminant function can be constructed for classification of two 
groups of cases, e.g., lymph node positive vs. lymph node negative, presence of distal 
metastases vs. none, local reoccurrence vs. none, death due to breast cancer vs. live, and poor 
prognosis (meaning any of the above occurrences) vs. good prognosis. 

In linear discriminant analysis, the discriminant function is formulated by a linear 
combination of the individual features. Linear discriminant analysis is a well-established 
statistical technique. Linear discriminant analysis is accomplished in two steps. First, a 
stepwise feature-selection is employed to select useful features to formulate the discriminant 
function. Second, the selected features are used to determine the coefficients in the 
discriminant function to achieve maximum separation. The criterion used to choose good 
feature variables in the stepwise procedure is accomplished by minimizing the ratio of the 
within-group sum of squares to the total sum of the squares of the distribution (Wilks 5 
lambda), i.e., a resubstitution test. A detailed discussion of the statistical theories for the 
stepwise procedure using the Wilks' lambda criterion is given in the literature [45]. 

Artificial neural networks are a non-algorithmic approach to information processing. 
Unlike many artificial intelligence techniques, which require extensive knowledge of the 
many parameters involved, ANNs learn directly from examples that are repeatedly provided. 
Once trained, a neural network can distinguish among input patterns on the basis of its 
learning experience. In the present invention, the analysis of the cases is performed using the 
image features and clinical data, and a "truth" value (e.g., 1 = lymph positive case and 0 = 
lymph node negative case). Multi-layer neural networks are used with one output unit. A 
back-propagation algorithm with generalized delta rule and Bayesian neural networks 
(BANNs) (19-21) may also be used. The input, which corresponds to the various breast 
image features/clinical data, is provided to the input layer of the neural network. The desired 
output (truth) is provided to the output layer during the training process. The hidden layer, 
which is important in the mapping of the input patterns to the output values, is located 
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between the input and output layers. A nonlinear logistic function is used as the activation 
function for each processing unit in the neural network. 

In the training process for the ANN, the internal parameters of the connections 
between layers (including threshold values of each unit) are adjusted iteratively so that the 
difference between the output values and the desired results is minimized. The output of an 
ANN in the limit of large sample sizes approximates a mapping function that is a monotonic 
transformation of the likelihood ratio, i.e., it estimates an ideal observer decision variable. A 
principal disadvantage of conventional ANNs is the over-parameterization of the mapping 
function, which results in a poor approximation of an optimal mapping function given small 
training sample sizes. Bayesian methods are applied to ANNs in order to regularize training 
to improve robustness. Ruck et al. [50] proved that an ANN trained using the sum of squares 
error function approaches the Bayes optimal discriminant function in the limit of infinite 
training data. Note that this maximum likelihood method approximates the sample Bayes 
optimal discriminant function when one has finite training data. It is necessary, however, to 
approximate the population Bayes optimal discriminant function because the sample Bayes 
optimal discriminant function is of little practical use. 

The classifiers used in the present invention can be trained using a resubstitution 
method and a jack-knife method, and performance in terms of area under the ROC curve (A z ) 
and partial -area-index [51] is examined as a function of classifier parameters such as number 
of training iterations, number of hidden units, and the learning rate. To test the capability of 
the classifiers to learn the image feature inputs, a resubstitution method was conducted in 
which the network was first trained with all the cases in the database, and then tested with the 
same cases used in the training. A resubstitution test indicates that the network is able to 
"remember" all of the input types that were used for training. 

However, it is more important to test whether the network can learn a generalized set 
of inputs from the examples provided, and if it can then make a correct prediction for new 
cases that were not included in the training. Thus, a jack-knife method is employed to test the 
network's generalizing ability. With a jack-knife method, either (a) all but one of the cases 
are selected randomly from the database for training of the network, and the remaining one 
case is used for testing the network; the output values are then compared to the "truth" data; 
or (b) half the cases are used for training and the other half is used for testing; then the 
database is randomly split in half again and the process is repeated. 

After a number of such analyses, the average A z value is computed from parameters 
of the ROC curves. Various combinations of training and testing pairs are selected by using a 
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random number generator and the results are analyzed using ROC analyses. ROC curves may 
be obtained by fitting continuous output data from the neural networks using the LABROC4 
program [52][53]. The area under the ROC curve (A z ) is used as an indicator of performance 
as well as the partial area index [53]. Once trained, the classifier will accept image features 
and will output a value that will be an estimated prognostic marker. 

In addition to the use of two-category discriminant analysis in the present invention, 
multivariate Cox regression models [54] may be fit to evaluate the prognostic value of the 
various summary measures derived from the imaging data after controlling for known clinical 
and pathological predictors, such as age at diagnosis and pathologic stage. The fit of the 
model should be examined using the graphical checks discussed in Kay [55], and through 
inspection of the Schoenfeld [56] residuals. Martingale residuals may be plotted to choose 
the appropriate functional forms for the covariates. After performing preliminary univariate 
analyses, the overall modeling-building strategy is to first construct a model that includes all 
important clinical and pathologic covariates, and then to sequentially add the (summary) 
imaging variables in a stepwise manner, since it is the added prognostic value of the latter that 
is of primary interest. After a final model is arrived at, it is validated using the bootstrap 
algorithm described in Harrel et al. [57] to obtain a bias-corrected estimate of predictive 
accuracy. 

First Prognosis Study 

In a first example study of the method of the present invention, the correlation 
between computer-extracted features of mammographic lesions with lymph node status was 
investigated. Figure 6 illustrates a result from the automatic lesion extraction on a 
mammogram. The margin of the lesion is shown. Figure 7 shows two example cases ~ one 
node positive and one node negative. The feature values are given for each case. For this 
comparison, the texture within the lesion contributes greatly to the prediction of nodal 
involvement. The features were merged using linear discriminant analysis, and then ROC 
analysis was used to assess the ability of the features to predict prognosis, via nodal 
involvement. 

Figure 8 shows the relationship between shape from radial gradient analysis and 
texture within the lesions for lesions from node-positive and node-negative breast cancer 
cases. The data points indicate the features for the mammographic lesions from node- 
positive cases, while the others indicate those from node-negative cases. 
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Figure 9 shows the relationship between the shape from radial gradient analysis and 
normalized texture within the lesions for lesions from node-positive and node-negative breast 
cancer cases. Here texture is the standard deviation of the gradients within the lesion divided 
by the average gradient within the lesion. The "*" data points indicate the features for the 
mammographic lesions from node-positive cases, while the others indicate those from node- 
negative cases. 

Figure 10 shows a histogram illustrating the distribution of node-positive and node- 
negative cases in terms of the output from the linear discriminant analysis. 

It should be noted that here the classifier to merge the lesion features was trained 
using nodal involvement. Training can also be performed using truth of other clinical 
prognostic factors such as presence of metastatic disease, local recurrence, death, and/or a 
combination. 

The output from the classifier is an assessment of prognosis for the case (patient). For 
example, the numerical value of the classifier output is related to a likelihood for survival, 
i.e., the aggressiveness of the tumor. It is expected that the computerized image analysis 
prognostic system will yield non-invasive prognostic indicators which can be used to aid the 
physician in decision-making regarding patient management and treatment. 

Note also that some computer-extracted features from the lesion are useful for 
diagnosis (malignant vs. benign) whereas other features are more useful for prognosis (good 
prognosis vs. poor prognosis). For example, while the computer-extracted spiculation feature 
described here (FWHM of the cumulative radial gradient histogram) is a strong feature for 
characterizing malignant (cancerous) lesions from benign (non-cancerous) lesions, it is not as 
strong in the task for assessing prognosis - good prognosis vs. poor prognosis. On the other 
hand, texture in terms of standard deviation of the gradients within the lesion region is 
stronger for prognosis than diagnosis. 

For prognosis, seventeen features were initially evaluated for their potential in 
assessing prognostic status of a cancer on mammogram. Four NRG (normalized radial 
gradient) features were evaluated, including: (1) NRG (margin) calculated along the margin 
of the lesion, (2) NRG (border) calculated in a widened border of the lesion, (3) NRG 
(within) calculated within the lesion, and (4) NRG (ROI) calculated within an ROI 
encompassing the lesion. Four FWHM features were evaluated including: (5) FWHM 
(margin) calculated along the margin of the lesion, (6) FWHM (border) calculated in a 
widened border of the lesion, (7) FWHM (within) calculated within the lesion, and (8) 
FWHM(ROI) calculated within an ROI encompassing the lesion. Additional evaluated 



22 



features include: (9) Average Gradient within the lesion, (10) Margin Sharpness (gradient 
along the margin), (11) Effective Diameter (diameter of a circle having the same area as the 
lesion), (12) Texture (standard deviation of the gradient within the lesion), (13) Ratio 
("Texture" divided by "Average Gradient"). Two contrast features were also evaluated: (14) 
the difference in gray levels between a center pixel and a band surrounding the boundary of 
the segmented lesion, and (15) the difference in average gray level within the segmented 
lesion and a band surrounding the lesion. The final two features evaluated were: (16) 
Circularity (the ratio of the portion of the segmented lesion overlapping the circle having the 
"effective diameter" centered at the lesion center to the area of the segmented lesion, and (17) 
Irregularity (determined mathematically as (1 - PERIMETERec)/PERIMETERsl; where 
PERIMETERec is the perimeter of the circle having the "effective diameter" and 
PERIMETERsl is the perimeter of the segmented lesion. 

The performance for the preferred features is given below in terms of Az value, which 
is the area under an ROC curve. Here the performance is evaluated in the task of 
distinguishing between good prognosis (node negative) and poor prognosis (node positive). 



Feature 


Az 


RGI(within) 


.67 


RGI(border) 


.64 


RGI(margin) 


.62 


RGI(ROI) 


.65 


FWHM(within) 


.66 


FWHM(border) 


.61 


FWHM(margin) 


.62 


FWHM(ROI) 


.65 


Effective Diameter 


.62 



The performance of the system was evaluated using ROC analysis (34) in assessing 
lymph node status on a database of 61 images from 43 patients with breast cancer, i.e., in 
distinguishing between cases with good and poor prognosis. Thirty-one images (24 patients) 
were lymph node negative and 30 images (19 patients) were lymph node positive. Using the 
three features (texture and shape), the system achieved an Az value (area under the ROC 
curve) of 0.82 in the task of distinguishing between node-positive cases and node-negative 
cases. Figure 1 1 shows Az values derived by merging two and three of the above-noted 
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features, the features being identified in Figure 1 1 . Performance improves as the number of 
relevant features being merged increases. 

The training of the classifier depends on the truth from the retrospective cases used in 
training the classifier, e.g., in determining the weights of an artificial neural network (ANN). 
Since the ultimate task of the computerized image analysis system is to assess the prognosis 
of a given cancer case (patient), the classifier needs to be trained using known prognostic 
indicators such as lymph node status (positive vs. negative nodal involvement), presence of 
local recurrence, presence of distal metastases, death, or a combination of prognostic 
indicators). An example of a combination is the categorization of cases into two groups - 
good prognosis and poor prognosis - in which a patient case would be placed in the poor 
prognosis category if any of the indicators was present, e.g., as long as there existed nodal 
involvement, local recurrence, distal metastases, or death. It should be noted that with the 
individual prognostic indicators and the combined method, the training is done using binary 
truth - good or poor prognosis. Another method of combining the truths is to indicate 
severity using a rating system, such as 1 if one of the indicators is present, 2 if two are 
present, and so on. Then the training of the classifier could be done with a "continuous" truth 
instead of a binary truth. Once trained, the computerized image analysis prognostic system 
will yield non-invasive prognostic indicators that can be used to aid the physician in decision- 
making regarding patient management and treatment. 

Second Prognosis Study 

In a second study of the method of the present invention, the correlation of lymph 
node status with computer-extracted features of mammographic breast lesions was examined. 
Eighty-one images from 51 breast cancer patients were collected along with the 
corresponding clinical information on nodal status. Thirty-three patients (48 images) were 
lymph node negative and eighteen patients (33 images) were lymph node positive. The 
mammograms were digitized at 12 bit (0.05 mm pixel size) and then subsequently 
subsampled to 0.1 mm. Locations of each cancer were manually identified by a radiologist 
and an ROI about each lesion was extracted. Given the center of each lesion, an automated 
method for lesion extraction and feature extraction was run. Fifteen computer-extracted 
features characterizing the lesion in terms of spiculation, margin, texture, and size were 
output and then merged using linear discriminant analysis. In addition, for each lesion, a 
corresponding parenchymal region on the contralateral breast behind the nipple was extracted. 
Twenty-three features characterizing the parenchyma were extracted. Selected features were 
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merged using linear discriminant analysis. ROC analysis was used to assess the ability of the 
features to predict nodal involvement. Using ROC analysis, the combined computer- 
extracted features of texture and radial gradient yielded an Az value of 0.80 (resubstitution) 
and 0.75 (round robin) in the task of separating into two groups: node positive and node 
negative. Combining the features with parenchymal features using linear discriminant 
analysis yielded an Az of 0.87 (resubstitution) and 0.80 (round robin). These results indicate 
the potential for the use of computer-extracted mammographic features in the assessment of 
cancer patients and their prognosis. 

Although the prognosis method of the present invention has been presented in the 
context of processing mammographic breast images, the method can be implemented using 
other breast images, e.g., sonographic breast images, in which a computerized image analysis 
is performed with respect to cancer or some other disease state. For example, using 
sonographic breast images, ultrasound lesions characteristics such as lesion margin sharpness 
and posterior acoustic characteristics can be used as lesion feature values in the method of the 
present invention. Moreover, the calculation of ultrasonic lesion characteristics is known in 
this art. See U.S. Patent. No. 5,984,870 (Giger et al„ Method and system for the automated 
analysis of lesions in ultrasound images), the contents of which are incorporated herein by 
reference. In addition, texture analysis on the sonographic parenchyma can be used as a 
parenchyma feature value in a manner analogous to the use of the parenchyma feature values 
described above for digital mammographic images. See U.S. Patent No. 6,282,305 (Huoet 
aL, Method and system for the computerized assessment of breast cancer risk), the contents 
of which are incorporated herein by reference. 

The present method can also be implemented more generally on other medical images 
of other organs (e.g., chest radiographs, or CT scans of the thorax, abdomen, or skeletal 
system) with respect to some other disease state or state of risk. Lesion and parenchyma 
feature values can readily be obtained from other medical images by those of ordinary skill in 
the art. For example, characteristics associated with a detected benign colonic region can be 
used as parenchyma feature values to modulate quantitative measures of malignancy 
calculated in methods of detecting cancerous polyps using colonographic images. The 
detection of such colonic polyps is discussed in co-pending U.S. Patent Application No. 
10/270,674 fYoshida et al. . Method for the computer-aided detection of three-dimensional 
lesions), the contents of which are incorporated herein by reference. Moreover, the detection 
of lesion feature values in various medical images is also well known in this art. See, e.g., U. 
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S. Patent No. 5,881,124 ( Giger et al.. Automated method and system for the detection of 
lesions in medical computed tomographic scans), the contents of which are incorporated 
herein by reference. 

For the purposes of this description an image is defined to be a representation of a 
physical scene, in which the image has been generated by some imaging technology: 
examples of imaging technology could include television or CCD cameras or X-ray, sonar, or 
ultrasound imaging devices. The initial medium on which an image is recorded could be an 
electronic solid-state device, a photographic film, or some other device such as a 
photostimulable phosphor. That recorded image could then be converted into digital form by 
a combination of electronic (as in the case of a CCD signal) or mechanical/optical means (as 
in the case of digitizing a photographic film or digitizing the data from a photostimulable 
phosphor). The number of dimensions that an image could have could be one (e.g. acoustic 
signals), two (e.g. X-ray radiological images), or more (e.g. nuclear magnetic resonance 
images). 

The present invention, while described above for the prognosis of breast cancer 
patients, can be used after appropriate feature calculation and classifier training for other 
modalities and cancer status assessment. For example, cancerous breast cancer lesions can be 
evaluated on x-ray (mammogram), ultrasound, and/or magnetic resonance images of the 
breast. Some features such as sharpness of the lesion margin (in terms of gradients) is a 
mathematical lesion descriptor that can be extracted on all three of these modalities. 
However some features are specific to the modality, such as the acoustic behaviour of lesions 
on ultrasound images and the uptake characteristics of contrast medium of lesions on 
magnetic resonance images. The feature of texture described here can be extracted across 
modalities. In addition, various lesion features are useful in characterizing lesions in other 
cancer sites. For example, the radial gradient analysis, the spiculation feature, and the texture 
in terms of standard deviation of the gradients within a lesion can be extracted on 
radiographic images of the chest to assess lung tumors. The same type of truth can then be 
used to train classifiers. Once trained, the computerized image analysis prognostic system 
will yield non-invasive prognostic indicators that can be used to aid the physician in decision- 
making regarding patient management and treatment. 

As disclosed in cross-referenced pending patent application 09/773,636, Figure 9 of 
that patent application is a schematic illustration of a general purpose computer 900 which 
can be programmed according to the teachings of the present invention. In Figure 9 of the 
cross-referenced application 09/773,636, the computer 900 can be used to implement the 
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processes of the present invention, wherein the computer includes, for example, a display 
device 902 (e.g., a touch screen monitor with a touch-screen interface, etc.), a keyboard 904, a 
pointing device 906, a mouse pad or digitizing pad 908, a hard disk 910, or other fixed, high 
density media drives, connected using an appropriate device bus (e.g., a SCSI bus, an 
Enhanced IDE bus, an Ultra DMA bus, a PCI bus, etc.), a floppy drive 912, a tape or CD 
ROM drive 914 with tape or CD media 916, or other removable media devices, such as 
magneto-optical media, etc., and a mother board 918. The mother board 918 includes, for 
example, a processor 920, a RAM 922, and a ROM 924 (e.g., DRAM, ROM, EPROM, 
EEPROM, SRAM, SDRAM, and Flash RAM, etc.), I/O ports 926 which may be used to 
couple to an image acquisition device and optional special purpose logic devices (e.g., ASICs, 
etc.) or configurable logic devices (e.g., GAL and re-programmable FPGA) 928 for 
performing specialized hardware/software functions, such as sound processing, image 
processing, signal processing, neural network processing, automated classification, etc., a 
microphone 930, and a speaker or speakers 932. 

As stated above, the system of the present invention includes at least one computer 
readable medium. Examples of computer readable media are compact discs, hard disks, 
floppy disks, tape, magneto-optical disks, PROMs (EPROM, EEPROM, Flash EPROM), 
DRAM, SRAM, SDRAM, etc. Stored on any one or on a combination of computer readable 
media, the present invention includes software for controlling both the hardware of the 
computer and for enabling the computer to interact with a human user. Such software may 
include, but is not limited to, device drivers, operating systems and user applications, such as 
development tools. Such computer readable media further includes the computer program 
product of the present invention for performing any of the processes according to the present 
invention, described above. The computer code devices of the present invention can be any 
interpreted or executable code mechanism, including but not limited to scripts, interpreters, 
dynamic link libraries, Java classes, and complete executable programs, etc. 

The programming of general purpose computer 900 (disclosed in cross-referenced 
pending patent application 09/773,636) may include a software module for digitizing and 
storing images obtained from film or an image acquisition device. Alternatively, the present 
invention can also be implemented to process digital data derived from images obtained by 
other means, such as a picture archive communication system (PACS). In other words, the 
digital images being processed may be in existence in digital form and need not be converted 
to digital form in practicing the invention. 
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Accordingly, the mechanisms and processes set forth in the present description may be 
implemented using a conventional general purpose microprocessor or computer programmed 
according to the teachings in the present specification, as will be appreciated by those skilled 
in the relevant art(s). Appropriate software coding can readily be prepared by skilled 
programmers based on the teachings of the present disclosure, as will also be apparent to 
those skilled in the relevant art(s). However, as will be readily apparent to those skilled in the 
art, the present invention also may be implemented by the preparation of application- specific 
integrated circuits or by interconnecting an appropriate network of conventional component 
circuits. 

The present invention thus also includes a computer-based product which may be hosted on a 
storage medium and include instructions which can be used to program a general purpose 
microprocessor or computer to perform processes in accordance with the present invention. 
This storage medium can include, but is not limited to, any type of disk including floppy 
disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAMs, EPROMs, 
EEPROMs, flash memory, magnetic or optical cards, or any type of media suitable for storing 
electronic instructions. 

Obviously, numerous modifications and variations of the present invention are 
possible in light of the above teachings. It is therefore to be understood that within the scope 
of the appended claims, the invention may be practiced otherwise than as specifically 
described herein. 
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